backup: fix AdminUnsplit race in online restore download cleanup#169758
Closed
msbutler wants to merge 2 commits intocockroachdb:masterfrom
Closed
backup: fix AdminUnsplit race in online restore download cleanup#169758msbutler wants to merge 2 commits intocockroachdb:masterfrom
msbutler wants to merge 2 commits intocockroachdb:masterfrom
Conversation
Contributor
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
|
This change is |
msbutler
force-pushed
the
butler-unsplit-err
branch
from
78d3dc8 to
34cec46
Compare
kev-cao
reviewed
May 7, 2026
Comment on lines
+823
to
+825
| besteffort.Warning(ctx, "or-unsplit", func(_ context.Context) error { | ||
| return errors.Wrapf(err, "failed to unsplit %s", sp) | ||
| }) |
Contributor
There was a problem hiding this comment.
This seems like a bit of a weird usage of besteffort.Warning. I think it'd be a bit more idiomatic to do:
besteffort.Warning(ctx, "or-unsplit", func(ctx context.Context) error {
if err := execCfg.DB.AdminUnsplit(ctx, sp.Key); err != nil && !kv.pb.IsKeyNotStartOfRange(err) {
return err
}
return nil
})| // IsKeyNotStartOfRange returns true if the error indicates that an AdminUnsplit | ||
| // request targeted a key that is not the start of a range. It falls back to | ||
| // string matching for compatibility with older nodes that produce the untyped | ||
| // error. |
Contributor
There was a problem hiding this comment.
Should we leave a TODO here to signal that the string matching can be removed in 27.1?
Introduce a typed sentinel error for the "key is not the start of a range" condition returned by AdminUnsplit. This replaces brittle strings.Contains checks with errors.Is, while including a string fallback in the IsKeyNotStartOfRange helper for mixed-version compatibility. Migrate the GC job's unsplitRangesInSpan to use the new helper. Epic: none Fixes: cockroachdb#169637 Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
A race between the GC job and the online restore download job could cause AdminUnsplit to fail with "key is not the start of a range" when the GC job merges ranges that the download job later tries to unsplit. This caused retry spiraling and eventual job failure. Make unstickRestoreSpans swallow "not start of range" errors (the range already merged, so the sticky bit is gone) and use besteffort.Warning for other errors, since unsplitting is cleanup that should not block the restore. Epic: none Fixes: cockroachdb#169637 Co-Authored-By: roachdev-claude <roachdev-claude-bot@cockroachlabs.com>
msbutler
force-pushed
the
butler-unsplit-err
branch
from
34cec46 to
7594580
Compare
|
Detected infrastructure failure (matched: self-hosted runner lost communication with the server). Automatically rerunning failed jobs. (run link) |
Collaborator
Author
|
going to try a different approach |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kvpb: add ErrKeyNotStartOfRange sentinel for AdminUnsplit
Introduce a typed sentinel error for the "key is not the start of a
range" condition returned by AdminUnsplit. This replaces brittle
strings.Contains checks with errors.Is, while including a string
fallback in the IsKeyNotStartOfRange helper for mixed-version
compatibility.
Migrate the GC job's unsplitRangesInSpan to use the new helper.
Epic: none
Fixes: #169637
Co-Authored-By: roachdev-claude roachdev-claude-bot@cockroachlabs.com
backup: make unstickRestoreSpans best-effort
A race between the GC job and the online restore download job could
cause AdminUnsplit to fail with "key is not the start of a range" when
the GC job merges ranges that the download job later tries to unsplit.
This caused retry spiraling and eventual job failure.
Make unstickRestoreSpans swallow "not start of range" errors (the
range already merged, so the sticky bit is gone) and use
besteffort.Warning for other errors, since unsplitting is cleanup
that should not block the restore.
Epic: none
Fixes: #169637